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ABSTRACT 

Missing data for a given cohort of students in a 
longitudinal study occurs for at least two reasons: either the 
student has moved or otherwise become unavailable for testing, or the 
cohort was not in the testing range at a given testing time. A 
developmental sampling for time of testing x cohort x grade research 
plan of testing is used to illustrate this point. K. W. schaie's 
(1965) proposed set of solutions addressing the developmental models 
of age, period, and cohort is considered. While the model is 
logically a three-way situation, computational/logical difficulties 
led to analyzing three two-way analyses, initially avoiding the use 
of missing cells and subsequently including missing cells. A model 
for time of testing x cohort x grade (age) research designs is 
proposed that follows the full three-way model. After a synthetic 
data set for grade, cohort, and period for standardized scores is 
constructed, linear models in a regression solution are applied. To 
analyze the data for cell, grade, cohort, and period, the sets of 
two-way design are addressed. The issue of the subjects effect is 
addressed after the two-way layouts are completed. The problem can 
then be viewed as a: (1) cohort x time of testing design, (2) grade x 
time of testing design, and (3) grade x cohort design. The issue of 
partial repeated measures and consideration of only the repeated 
measures are discussed. Alternative hypotheses are evaluated, 
including a tine of testing x cohort x grade design with partial 
subject control with weighted and unweighted means hypotheses. Ten 
data tables illustrate the different designs. A 17-item list of 
references is included. (RLC) 
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Developmental Models for Time of Testing 
x Cohort x Grade (Age) Research Designs 

John Delane Williams 
University of North Dakota 

When longitudinal data are addressed , problems with 
missing data often occur. Missing data for a given cohort 
of students will occur for at least two reasons: either the 
student has moved or otherwise become unavailable for 
testing, or the cohort was not in the testing range at a 
given testing time. 

A diagram is shown in Table 1 that would reflect the 
research design. 

Table 1 

Developmental Sampling for Time of Testing 
x Cohort x Grade Research Designs 

Time of Testing 
Cohort 1985 1987 1989 1991 

Grade 6, 1985 cell 1 

Grade 4, 1985 cell 2 

Grade 2, 1985 cell 3 

Grade 2, 1987 cell 6 cell 8 cell 10 

Grade 2, 1989 cell 9 cell 11 

Grade 2, 1991 ce ll 12 

The plan of testing is each second, fourth and sixth grade 

would be tested every other year, so that in a given year, 

three grades would be tested. At any given time, some 

groups would have only been tested once, while others would 

have been tested either two or three times; missing cells 

would occur quite logically. 



Time of 


Testing 


1987 


1989 


cell 4 




cell 5 


cell 7 


cell 6 


cell 8 




cell 9 
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Background of the Design 

Scheie (1965) proposed a set of solutions that 
addressed the developmental models of age, period and cohort 
with specific interest in aging research. While the model 
is logically a three-way situation, computational/ logical 
difficulties led to analyzing three two-way analyses, 
avoiding the use of missing cells. Thus only a subset of 
the entire data set would be used in various portions of the 
analysis. While there were many critics of Schaie's 
original work, that seminal effort is such that it is still 
in use today. 

Baltes* (1968) criticisms helped lead to the Schaie and 
Baltes (1975) compromise emphasizing the age x cohort 
layout. Each of the two-way layouts has side restrictions 
(assumptions) regarding their use. Also, at least two of 
the three two-way layouts would include missing cells. The 
missing cells issue was circumvented by truncating the 
analysis with the deletion of cells until no missing cells 
occurred. This was done in such a way as to minimize the 
loss of cells. Missing cell solutions are still only 
infrequently encountered in published research . 

Schaie (1977) has addressed certain designs ( 2 x 2 or 2 
x 2 x 2) with missing cells, developing specialized 
processes for each. Palmore's (1978) model (a 2 x 2) is 
drawn with a missing cell. A full three-way model was 



recently developed by Williams (in press). The present 
approach follows from that full three-way model. 

A Synthetic Data Set 

First, a synthetic data set is constructed for a 
situation similar to, but somewhat less complex than the one 
shown in Table 1. Let us assume, for the moment, that 
testing began in 1989 and a second test was administered in 
1991; see Table 2. 

Table 2 

Developmental Sampling for Period x Cohort 
x Grade for Two Testing Periods 

Conort 

1 . Grade 6 , 1989 

2. Grade 4, 1989 

3. Grade 2, 1989 

4. Grade 2, 1991 

For purposes of illustration, cohort 1 is assigned 4 
members, cohort 2 is assigned 5 members, cohort 3 is 
assigned 6 members and cohort 4 is assigned 7 members. 
Clearly, actual class sizes will usually be much larger; the 
smaller groups are convenient for illustration, however. 
See Table 3. 



1989 

cell 1 
cell 2 
cell 3 



1991 



cell 4 
cell 5 
cell 6 
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Synthetic 


Data for Grade, Cohort and 






Period 


for S tandarai zed 


Scores 
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1 AAA 

1989 




1991 
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42 




6 


42 
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46 
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39 
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o 
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35 
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52 




48 


o 
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1 1 


55 




49 
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1 2 


51 




47 


o 
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13 


53 




43 


6 


1 A 

14 


48 




49 


3 


15 


42 




43 
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16 






56 


4 


17 






58 


4 


18 






62 


4 


19 






60 


4 


20 






57 


4 


21 






55 


4 


22 






48 



The data in Table 3 represent a test whose population mean 
for any grade is 50 and whose standard deviation (for the 
population) is 10. The approach uses linear models in a 
regression solution (Ward & Jennings, 1973; Williams, 1974) 
characteristic variables (1 or 0) sometimes called binary 
variables, are used. See Table 4. Subject coding is 
accomplished in the manner described by Pedhazur (1977) and 
Williams (1977b). For persons not retested, no subject 
totals are shown. 
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Table 4 

Synthetic Data for Cell, Grade, Cohort, and Period 

Grade Cohort Period Subject 



Scores 


1 


2 


3 


4 


5 


6 


2 


4 


6 


1 


2 


3 


4 


1989 


1991 


Total 


43 


1 


0 


0 


0 


0 


0 


0 


0 


1 


1 


0 


0 


0 


1 


0 




42 


1 


0 


0 


0 


0 


0 


0 


0 


1 


1 


0 


0 


0 


1 


0 




36 


1 


0 


0 
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0 


0 


0 


0 


1 


1 


0 


0 


0 


1 


0 




33 
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0 


0 
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0 
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0 


1 


1 


0 


0 


0 


1 


0 




45 


0 


1 


0 


0 


0 


0 


0 


1 


0 


0 
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0 


0 


1 


0 


87 


42 
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1 


0 


0 


0 


0 


1 


0 


1 


0 


0 


0 


1 


87 


42 


0 


1 


0 


0 


0 


0 


0 


1 


0 


0 


1 


0 


0 


1 


0 


82 


40 


0 


0 


0 


1 


0 


0 


0 


0 


1 


0 


1 


0 


0 


0 


1 


82 


46 


0 


1 


0 


0 


0 


0 


0 


1 


0 


0 


1 


0 


0 


1 


0 


84 


38 


0 


0 


0 


1 


0 


0 


0 


0 


1 


0 


1 


0 


0 


0 


1 


84 


39 


0 


1 


0 


0 


0 


0 


0 
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0 


0 


1 


0 


0 


1 


0 
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36 


0 


0 


0 


1 


0 


0 


0 


0 


1 


0 


1 


0 


0 


0 


1 


75 


35 


0 


1 


0 


0 


0 


0 


0 


1 


0 


0 


1 


0 


0 


1 


0 


67 


32 


0 


0 


0 


1 


0 


0 


0 


0 


1 


0 


1 


0 


0 


0 


1 


67 
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0 
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0 
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0 


0 
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0 
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0 
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0 
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0 


0 


0 
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0 
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104 


51 
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0 


1 


0 


0 
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0 


0 
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0 


1 


0 


98 


47 


0 


0 


0 


0 


1 


0 


0 


1 


0 


0 


0 
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0 


0 


1 


98 


53 


0 


0 
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0 


0 


0 
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0 


0 
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0 
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43 
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0 
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0 


0 
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1 
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0 
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97 
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1 
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0 
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42 
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0 
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85 


43 
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56 
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0 
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1 





Note that subject totals are not recorded for persons 
in either the first or fourth cohort; persons in these 
cohorts have only been tested once and could not (at this 
point) allow for subject control. This circumstance is also 
different from that shown in Williams (in press) which was a 
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repeated measures design. To effect a solution f it is 
useful to define several variables which have been shown 
Table 4: 

Y = the standard score; 

Xi = 1 if the score is from a member of cell 1» 0 
otherwise ; 

Xa = 1 if the score is from a member of cell 2, 0 
otherwise ; 

X3 = 1 if the score is from a member of cell 3, 0 
otherwise ; 

X 4 : 1 if the score is from a member of cell 4, 0 
otherwise ; 

X* = 1 if the score is from a member of cell 5, 0 
otherwise ; 

Xe = 1 if the score is from a member of cell 6, 0 
otherwise ; 

X7 = 1 if the score is from a member of grade 2, 0 
otherwise ; 

X$ = 1 if the score is from a member of grade 4, 0 
otherwise ; 

X* r 1 if the score is from a member of grade 6, 0 
otherwise ; 

X10 = 1 if the score is from a member of cohort 1, 0 
otherwise ; 

Xn : 1 if the score is from a member of cohort 2, 0 
otherwise ; 



Xji = 1 if the score is from a member of cohort 3, 0 
otherwise ; 

Xi 3 = 1 if the score is from a member of cohort 4, 0 
otherwise ; 

Xi « = 1 if the score is from a member tested in 1989, 0 
otherwise ; 

Xi 5 = 1 if the score is from a member tested in 1991, 0 

otherwise; and 
Xi 6 = the sum of scores for a given member. 

To analyze the data, first, the sets of two-way designs are 
addressed. The issue of the subjects effect is addressed 
after the two-way layouts are completed. 

Viewing the Pr oblem as a Cohort x Time of Testing Desi fln 

For cohort x time of testing, Table 2 shows the layout 

of the cells. The cohort effect can be found by using 

HI Y = b 0 + bioXio + bnX,, + bi 2 Xi, + ei , 

which yields Ri * = .730200; SScoiobt = 1493.63. This 

solution will correspond to testing the hypotheses 

u » = m ui + n4 Uj - naU3 + rnu } = u e . 
n 2 + n« n$ + ns 

These hypotheses can be tested against the full model: 
[2 ) Y = bo + biXi + b 2 X 2 + b 3 X s + b« X 4 + b 3 X 5 + e 2 . 

For equation 2 , R 2 2 = . 76756; SS T = 2045.51, with SSchls = 
1570.05. The full model without an intercept term is given 
by 

[31 Y = biXi + b 2 X 2 + b 2 X| ♦ b«X« + b 8 X, ♦ beX« + e 2 . 

7 



The time of testing effect is measured by 



[4] 



Y = 



bo +bi4Xi« + e 3 ; 



Rtine 2 = .05810, SStime = 118.84. 

To find the cohort x time of testing interaction, it should 
first be noticed that the missing cells will be reflected in 
testing for interaction. The two missing cells will cause 
the degrees of freedom for interaction to be one rather than 
three. The single hypothesis for interaction can be stated 
as uz - us s U4 - us . In terms of the regression 
coefficients, the hypotheses can be tested by 
b2 - b3 = b4 - bs , or b 2 = b# + b3 - bs . Placing this 
restriction on the full model (equation 3) yields 

Y = biXi + (b 4 + b 3 - b 5 )X 2 + b 3 X 3 + b 4 Xt + bs Xs + b» Xe + 
e 4 , or 

Y = biXi + b 3 (X 3 + X 3 ) + b4(X 4 + X 2 ) + b 5 (X 5 - X 2 ) + b 6 Xe + 



Letting Vj = X 3 + X 2 , V 2 = X 4 + X 2 and V 3 = Xs - X 2 , the 
previous equation becomes 

Y = bi Xi + b 3 Vi + b4 V 2 + bs V 3 + b« X« + e 4 ; reparameter i zing 

by letting one of the b's equal 0 (say b« ) yields 



The use of equation 5 yields R 5 2 = .76755, SSs = 1570.03. 
The sum of squares for interaction will be SS T = 1570.05 - 
1570.03 = .02. An alternative, and simpler way, to address 
the interaction issue is to use 



e 4 . 



[5] 



Y = bo + biXi + b 3 Vi + b«V 2 + b 5 V 3 + e« . 



[6] 



Y = 



bo + bioXio + biiXii + bnXn + b 24 Xi 4 + e 4 ; 
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where Xio, Xi i and X» i are variables that show membership 
respectively in cohorts 1, 2 and 3 (cohort 4 is then simply 
not a member of cohort 1, 2 or 3 ) and Xi « is a binary 
variable showing time of testing. The results for R* and 
the sum of squares for the use of equation 6 are identical 
to equation 5. 

Viewing the Pr oblem as a Grade x Time of Testing Design 

Table 5 shows the layout when the problem is viewed as 
a grade x time of testing design. This particular layout 
has no missing cells. 

Table 5 

Grade x Time of Testing Layout 
Grade 1989 1991 

Grade 6 cell 1 celI 4 

Grade 4 cell 2 cell 5 

Grade 2 cell 3 cell 6 

While the layout will often involve disproportionate data 

together with the various interpretations associated therein 

(see, for example, Applebaum and Cramer, 1974; Cohen, 1968; 

Jennings, 1967; Overall, Spiegel and Cohen, 1905; Timm and 

Carlson, 1975, and Williams, 1972, 1977a) this portion of 

the analysis is at least not also burdened by missing cells. 

The sum of squares for time of testing were previously 

found; thus only the grade effect and grade x time of 

testing effect need additionally to be found at this point. 

The grade effect is measured by 
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[7] Y = b 0 + b 7 X7 + baXg + es . 

The use of equation 7 will correspond to testing the 
hypotheses 

niUi + n«ui = ni U| ± n^u g = nj ui + nj u« . 
ri\ -¥ th ni + ns ni ♦ n« 

For equation 7 , R 7 1 = .66722 and SSeuDt = 1364.80. To 

find the grade x time of testing interaction, the difference 

between the full model (equation 3) and a model including 

X7 , Xg (grade) and Xj 4 (time of testing) is found. 

[8] Y = bo + b*X 7 + b 8 X. + bi 4 Xi 4 + e. . 

For equation 8 , R« » = .73073, SSs = 1494.72. Then 

SSqiade x time = 1570.05 - 1494.72 = 75.33, which is also 

equal to (R 2 1 - R« 2 )SS T . 

Viewing the Problem as a Grade x Cohort Problem 

Table 6 shows the layout when the problem is viewed as 

a grade x cohort design. Note the number of missing cells. 

Table 6 

Grade x Cohort Layout 

Cohort Grade 2 Grade 4 Grade 6 

Grade 6, 1989 cell 1 

Grade 4, 1989 cell 2 cell 4 

Grade 2, 1989 cell 3 cell 5 

Grade 2, 1991 cell 6 



Inspection of Table 6 would allow the inference that, for 
the present design, there is no grade x cohort interaction 
that is testable. If a model for grade and cohort were 
attempted 
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[9] Y r bo + D7X7 + btXt + bioXio ♦ buXn + bitXn + et, 
then Rt 1 = .76756 and SS 9 = 1570.05, the same results that 
were obtained for the full model (or cell model, equation 
2). Also, there are no degrees of freedom remaining for the 
grade x cohort design. Had more than two time of testing 
periods been used, then there would exist testable 
hypotheses regarding interaction on a limited part of the 
grade x period layout. 

If the various parts of the outcomes are gathered into 
a table (see Table 7), the degree of overlap of the 
variables becomes even more apparent. 

Table 7 

Outcomes from Time of Testing x Cohort x Grade Design 



Source of Variation 


df 


SS 


MS 


F 


Time of Testing 


1 


118.84 


118.84 


6.75 


Cohort 


0 


1493.63 


497.88 


28.27 


Grade 


2 


1364.80 


687.40 


39.03 


Time of Testing x Cohort 


1 


.02 


.02 


.00 


Time of Testing x Grade 


2 


75.33 


37.66 


2.14 


Within 


27 


475.46 


17.61 


Total 


32 


2045.51 






Not only are the sums of 


squares , 


when totaled (SSt 




3528.08) greatly larger 


than SSt 


- 2045.51 


, but the 


sum of 


the degrees of freedom, 


36, is larger than 


the total 


degrees 



of freedom, 32. The likelihood of confounding with missing 
cells is typical; what the researcher chooses to do, 
however, would seem to be directed by the choice of 
hypotheses that are viewed as appropriate for testing. 
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Addressing the Issue of Partial Repeated Measures 

Of the four cohorts, two have two measures per subject, 
and two cohorts have only a single measure . This situation 
presents some difficulty, and an equation of the form 
[10] Y = bo + bitXig + e7 

is not appropriate for capturing the subject effect , because 
Xi«, by itself, would also contain information regarding the 
nontested cohorts; in conjunction with the other cells, the 
use of the Xi« would allow the calculation of the subjects 
effect. See equation 11. 

C onsidering Only the Repeated Measures 

If only the two classes that are retested at the second 
time period are included, then a simple repeated measure 
design results. Such a design can be found either through a 
regression methodology (see, for example, Williams, 1974) or 
through several other computer packages, including SPSS* and 
SAS. Table 8 shows the summary table for this analysis. 
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Table 8 

Summary Table for the Repeated Measures Design 



Source of Variation df 


SS 


MS 


F 




Among Subjects 10 
Cohorts 1 
Erron 9 


656.27 
425.61 
230.66 


425.61 
25.63 


16. 


61 


Within Subjects 11 
lime ot lestmg l 
Cohort x Time of Testing 1 
Error2 __9 


132.50 
76.41 
.02 
56.07 


76.41 
.02 
7.23 


10. 

• 


0 ( 

00 


Total 21 


787. 77 








Two parts of Table 8 are useful in 


constructing a table 


that 




includes the two analysis reported 


earlier 


in Table 7 with 




the repeated measures information. 


The sum 


of squares 


for 





interaction is the same in both tables; this is because the 
interaction addresses only the subjects involved in the 
repeated measures* The sum of squares errori (the error 
among subjects) captures the variation due to subject 
control* A summary table including this source of variation 
is shown in Table 9 . 



13 



15 




Table 9 

Time cf Testing x Cohort x Grade Design with 
Partial Subject Control 



oource 


oi variation 


at 


bS 


MO 

MS 






r 


Time of 


Testing 


1 


118.84 


118 


.84 


8 


.74 


Cohort 




3 


1493.63 


497 


.88 


36 


.61 


Grade 




2 


1364.80 


687 


.40 


50 


.54 


Time of 


Testing x Cohort 


1 


.02 




.02 




,00 


Time of 


Testing x Grade 


2 


75.33 


37 


.66 


2 


.77 


Error - 


Within 


27 


475.46 


17 


.66 






Errori 


- Subjects 


9 


230.66 










Error* 


- Within Subjects 


18 


244 .80 


13 


. 60 






Total 




32 


2045 . 51 











Another way to try to isolate the portion of the withi 1 
error term due to subjects is to use the model 
[11] Y bo ♦ biXi + baXa ♦ baX» + b«X« ♦ b 5 X 5 + bieXie 

+ es . 

Equation 11 will yield Ri i 2 = .88035 or SSi i = 1800.67. 
Then, SSi i - SS 2 = 1800.67 = 1570.05 = 230.62. This value 
is close to erron in both Tables 8 and 9 (230.66). 

Other Hypotheses of Possible Interest 

Researchers who prefer the "full rank model" solutions 

of Timm and Carlson (1975) would prefer to test hypotheses 

somewhat different then those shown thus far. The 

hypotheses shown for testing the cohort effect, 

ui = QUil i n4 yj = nj_uj ± naus = ue , 

nz + n4 n$ + ns 

will correspond to the hypotheses that would be tested by 

the approach suggested by Timm and Carlson (1975; but also 

see Williams, 1977a, in press), since n» = n 4 and n$ = n 5 . 
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For nonrepeated measures designs, this would not always be 
the case. 

The time of testing effect already shown had as its 
tested hypothesis 

niui + n2 U2 + mua = nj Uj + ns Us + ne ue . 
ni + n2 + n 3 ne + ns + ne 

The corresponding "full rank model" hypothesis is 

ui + U2 + U3 = ue + us + ue , or in terms of the regression 

coefficients, bi + b 2 + b 3 = be + b 5 + be or hi = b« + b 5 + 

be - b2 - b 3 . If this restriction is substituted into 

equation 3, then Y = ( be + b 5 + be - b 2 - b 3 )Xi + b 2 X 2 + 

baX3 + be X4 + bsXs + be Xe + e 9 . Rearranging in terms of the 

regression coefficients, Y = b 2 (X 2 - Xi ) + b 3 (X 3 -Xi ) + 

b 4 (Xe + Xi ) + b 5 (X 5 + Xi ) + b»(Xe - Xi ) + e« . 

Reparameterizing by letting any bi = 0 (say be) yields 

Y = b 0 + b 2 (X 2 - Xi ) + b 3 (X 3 - X, ) + be ( Xe + Xi ) + b 5 (X 5 + 

Xi ) + e. . Letting V 4 = X 2 - Xi , V» = X 3 -Xi , V e = X 4 + Xi 

and V 7 s X 5 + Xi yields: 

[12] Y = b 0 + biVe + b 2 V s + b 3 Ve + be V 7 + e 9 . 

The use of equation 12 yields Ri 2 1 = .71875 and SSi 2 = 

1470.21. To find the "full rank model" measure of the time 
of testing effect, SS a - SSi 2 = 1570.05 - 1470.21 = 99.84, 
which is slightly lower than that found earlier for the time 
of testing effect. 

The grade effect has as its hypotheses (using the Timm 
and Carlson [1975] approach) 
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ui + U4 = u» + us = ua t ut , or in terms of the regression 
2 2 2 

coefficients, bi ♦ b 4 = bi + b 5 = bs + b« or bi = ba + bs - 

b 4 and ba = ba + bs -be. Substituting these coefficients 

into equation 3, 

Y = (ba + b s - b 4 )Xi + ba Xa + ( ba + bs -b 6 )X 3 ♦ b« X« + b 5 X 5 
+ be Xe + ei o » or 

Y = ba ( Xa ♦ Xi + X 9 ) + b 4 (X 4 - Xi ) + bs (Xs + Xi + X 3 ) + 
b6 { Xs - Xa ) + ei o . 

Letting b« = 0 and since Xi + Xi + X 3 = Xi « , we can let Vs = 
X 4 - Xi , and let V» = X 5 + Xi + X 3 , yielding 
[13] Y = bo + b s Xi 4 + b 4 V 8 + bs V t + eio. 

The use of equation 13 yields Ri 3 2 = .12750 and SSj 3 = 
260.18. Thus, the grade effect for the Timm and Carlson 
(i.e., unweighted means) approach is SSa - SSi 3 = 1570.05 - 
260.18 = 1309.24. This value is relatively close to the 
weighted means approach found earlier of 1364.80. A final 
summary table including the entire outcomes of this design 
are shown in Table 10. 
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Table 10 



Time of Testing x Cohort x Grade Design with 
Partial Subject Control with Weighted and 
Unweighted Means Hypotheses 



Source of Variation 


df 


SS 




MS 




F 


Time of Testing (Weighted) 


1 


118. 


84 


118 


.84 


8. 74 


Time of Testing (Unweighted) 


1 


99. 


84 


99 


.84 


7.34 


Cohort 


3 


1493. 


63 


497 


.88 


36.61 


Grade (Weighted) 


2 


1364 . 


80 


687 


.40 


50.54 


Grade (Unweighted) 


2 


1309. 


24 


654 


.62 


48.13 


Time of Testing x Cohort 


1 


• 


02 




.02 


.00 


Time of Testing x Grade 


2 


75. 


33 


37 


.66 


2.77 


Error - Within 


27 


475 . 


46 


17 


.66 




Erron - Subjects 


9 


230. 


66 


25 


.63 




Error2 - Within Subjects 


18 


244. 


80 


13 


.60 




Total 


32 


2045. 


51 









In evaluating the outcome of such an experiment, many 
different strategies can be employed—one is suggested here* 
When the outcomes of both the weighted and unweighted 
hypotheses are the same, a conclusion can then be drawn; if 
the outcomes are different, then the results are deemed 
inconclusive (this follows the rationale of Applebaum and 
Cramer, 1974)* If testing is conducted exclusively at the 
.01 level, there is a difference in outcome for the time of 
testing regarding the weighted (p < *01) and unweighted («01 
< p < ,05) tests. Some researchers might prefer using the 
MSw rather than MSe*ror 0 as the error term. 
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